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This paper describes design techniques for generative computer-assts+ed 
Instruction (CAI) systems. These are systems which are capable of generating 
problems for students and deriving and monitoring the solutions to these 
/problems. The difficulty of the problem, the pace of Instruction, and the , 
depth of monitoring are all tailored to Hie Individual student. Parts of 
the solution algorithms can also be used 'to analyze an incorrect student 
response and determine the exact nature of the students error In order to 
supply him with meaningful remedial comments. V = 

v: A generative CAI system which teaches logic design and machine- 
language programming will be discussed. This CAI system covers the matenj^li 
In an introductory course I n digital system aimed at electr leal engl neerlng 
Juniors. It does not replace classroom lectures or the textbook,, but Instead 
serves to provide practice and instruction In applying this materia I to solve 
" terns. . , > • . 



In addition, a companion system to teach laboratory principles has been 
designed. This system teaches a student hew to construct a combinational or 
sequential logic circuit us I ng standard integrated circuits. The student f s 
logic circuit is automatically interfaced tc the computer and tested; the . 
computer then aids the student in debugging his circuit. 



Work in progress on the design of a tutor for high-school 



which 



i 



ra word problems, is also described. 



teaches students how to solvqjg] 

Final ly, a formal mathematical approach to problem generation and solution 
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I. INTRODUCTION 

Over the past two years a set of three computer-ass! sted Instruction;. 
(CAD systems has been designed around an introductory course In computer 
science. Thjs course Is taught In the Electrical Engineering Department at 
the University of Connecticut and Is required of all Electrical Engineering 
students. It is also taken by students In other departments who wish to minor 
In computer science. J / _ ^ 

The first system, COMSEQ, (Koffman, 1972) Introduces students to the 
design and simp I if Icatlon of COMbi national and SEOuentla I digital logic circu its. 
It starts off by teaching the binary, octal, and hexadecimal number systems 
and logical and arithmetic operations within these systems. It covers the 
design of combinational circuits and the Karnaugh Map and QuIne-McKIuskey t 
methods of minimization. Final ly, it teaches how to use Fllp-Flcps In the 
analysis and synthesis of sequential logic circuits. 

The second CAI system, MALT, (MAchine Language Teacher - Blount and 
Koffman, 1972) teaches stucents how to program a minicomputer using machine 
language. The computer used is similar to the Digital Equipment Corporatl :>n 
PDP-8. For si 



I city, the MALT version has on ly 377 (octal) words of core. 



t 



The third" system, CAI ID, (Computer Aided Instruction In Logic Debugglng- 
Wel, 1973) Instructs students In the design and debugging of digital logic 
circuits. The circuits are built by the student on a special logic board 
which contains TTL Integrated circuits. These Include NANO gates, Inverters, and 
JK and D Flip-Flops. Sequential circuits with two external Inputs and outputs 
and up to sixteen states can be constructed. Combinational circuits with up 



I 
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These systems a re or I ented toward s teach I ng prob 1 em so I v i ng a nd ass Pst 
the student In learning how to apply the concepts introduced In class or_ the 
textbook (Booth, 197 1 ). The f i rst two systems have been used extensively 
and appear to serve very wdllas a replacement for coventlcnal homework. 
They have the advantage of easing a beginning student Into the problem and 
providing prompt' and perHncnt remedial feedback when he goes astray. Both 
systems are programmed on the IBM 360/65 and are aval ! able through the CPS 
(Conversational Programming System) time-sharing system. Students can ca 1 1 
these programs from any of the forty CPS term Inals cn campus and continue 
with their CAf work whenever they wish. 

Both systems are fully "generative". This means that the students do 



not work "canned 1 * or pre-stored problems. The system generates, for each 



student, problems which are Individualized with respect to his previous per- 



formance In the CAI course. The problems generated are normal ly not too 



I 
I 



d If f Icu I t nor too easy and are different from other problems attempted. The 
generation process does not consist of merely plugging randomly generated 
parameters into preset question formats; rather, the problem format Itself 
Is often constructed from a set of basic problem elements. This provides for 
a wide variety bif/S|j^ 

g^^^K|^?^^^Kua| problem presented Is not predetermined, neither Is the 

^j^^ ai^^th- 
J^jj^Si^^ _tge J^Jbn 

^gnfr fl^^^ _;on h Is p:ge^1g) u s 

jigj^gr^^ dtj^h&ncit whereas; COMSEQ: ajnd^ 

^:pj^gr|^I^^ ;fbr 7a^ 
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to concentrate on the more cemp I ex aspects of : tbe problem. 

The third system, CAILD Is capable of degugglng any circuit which the 
student has designed;. He must specify the logical equations which describe 
his circuit In Sum-of-Product form. The student_ logic beard Is automatically 
: Interfaced to the Dig I tal Equipment Corporation PDP-9 computer which then 
controls the debugging process. " = . 

The emphasis is cn teaching good debugging procedures rather than automa- 
ting fault detection. Circuit Inputs and Fl Jp-Flop state values are control led 
by CAILD. Critical output logic values are monitored. In addition, the 
student Is directed to utilize a test-probe to monitor suspicious points In 

Information with Its model of the student f s circuit to help him trace an error 
to Its source. CAILD also allows a student to test his circuit after It has ■ 
been debugged by applying current input and state conditions and displaying 
the next state and output values. ^fy^^^^^^f^^^S^ r ^^j^^jjjv 
Taken together, these three CAI systems comprise an Integrated package 
of programs which cover most aspects of logic design and machine language 
programming. The intent is not so much to instruct a student In the basic 
concepts but to guide him In applying what he has learned to solve problems, 
write programs, or debug and test c I rcu I ts5L£2^ 

Chapters two through five describe these systems and their application 
Stpf^he cou r se . Chapter six descr I bes a^tutb^fb^ 
design Is strongly influenced by a||£E^ 

v^ylJemccan h^ grapJiJng^vV, 
?p(^l^^: ^ .-- v ~ 
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abllity; manipulation problems test his -grasp of the fundamental skills and 
techniques; graphing problems test his basic understanding of graphic methods 
and abil Ity to plot equations. 

Chapter seven Is a fbrrna! model for problem generation and solutlbn.- 
Probiem generation: Is described as orlginati ng from a semantic network 
(Qui j I Ian, 1969). This process is an extension of Carboriel I *s work to 
quantitative problem solving (CarboneM, 1970) and Is motivated by Polyafts 
classic work on problems (Polya, 1945). Indeed, this research"can be thought 
ibf as a step In the automation of Polya's heuristics. Problem solution 
Is closely related to problem generation. The plan for problem sol utlon Is 
generated at the same time that the problem Is generated. 
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the JCOMSEQ System 

fA;.-= Overview . .-2, if. 

= The system Is designed to be extremely f lexible. In that It can completely: 
control the. progress qf a student through- the course, selecting concepts fctr..- r ^ 
Jljudy on an Individual basis and generating problems. Alternatively, the 
^Judent esjn assume the Initiative and determine his own areas of study andZe^-. 
supply his own problems. -r : - 1 _ . 

--. In addition, the system also operates In a ''problem-sblye^mcde. IjO^K? 
this mode, the student specifies the concept area and his problem, and the 
system will crank out the solution without further interaction. It Is \ 
anticipated that students la later courses and the dlglta I laboratory wi 1 1 



i 

i 



utilize this mcde for solving complex minimization problems and determining 



the relative merits of different state assignments. 



Figure I is a block diagram of the system functions. Subsequent sections 



ggf^thls paper will describe how these functions are accomplished. As has been 



mentioned, the student can assume the initiative and by pass the Concept 



Selector and/or Problem Generator (Indicated by dashed lines from Student 
box). The bottom part of Floure 1 shows the student exercising full control 
In the problem solver mode. 

B. Concept Selector ^^^^^^-^^^^^^r ' •, - . - 

the system Is In centre! 'of t^ffrt®6sggIon^It a^Mp^s iO: -: ^ 



Indlvldua 1 1 ze the depth and pace of Instruction presented to each student. A 
^^^L^^ch student Is kept which summarizes his past performance. In each 
jgl^he course concepts. Table I shows the contents c a student record. 7^5 
|^ A student f s LEt|pgii^rea I number b^^^^^o^ eye! ^ 

tfe concept to solve a 
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__- _ = Figure I 
System Blcfclc OTiagram 



; TABLE I— Student Record - ~ -> ~ 

Student Name E B Kofrman Master Ave. LS Current Plnteau 5 
^ v; Concept #L Concept £2 . . . Concept »30 



i 



/Level- :="--" 


2.2 


1.5 - 


V--' 0.5- 


Last?! eve! change 


a 


.5 


-.2 


Weighted Av. level 




,G 




> c^ angeir : _ 








patf of^a^ticalt 


3/15 


3/17 


4/20 


Sequential order of 


25 


--, 31 


<1S - 


las _^_call"-; r --- " ? 








No^of f times fcallect. 


2 / 


2 


3 


: ihiC^l-range^ 
NoVofitimeV called 


_ 2 


1 


0 


-in 1-2 range/ 








No. of times called 


0 . 


0 


0 


: in 2-3fanfe 








^o; of problems 


2 


3 . 


« o 


generated 









I 

I 

1 



1 



I 1 II 
Ie 1 m 



problem. A small Increase In LEVEL (.07 to .28) occurs for each correct 
an c <er and a smalt decrease for each Incorrect answer, the student's LEVEL 
"fcn a conce .te ^ilnes the difficulty of the problem generated and the 
amount of Interaction he .receive^ durlng= I ts= solution. In calculating the 
weighted average level change, the most recent change In a concept's LEVEL 
has the greatest effect. Normal ly, the number of problems generated fn a 
concept area for a student who Is performing well is around three or four. 

In addition, the system Is supplied with a concept tree which indicates 
the degree of complexity (plateau) of a concept and Its relationship with other 
concepts In toe course. A sample tree for the concepts currently covered _ " 
Is shown in Figure 2. This represents the author 1 s Interpretation of the 
relatl on ship between course concepts. There are alternate Interpretations 
which are just as valid. - } * " V ^ , - 

' The system uses each student's record to determine how qu Ickly he shou Id 
progress through the tree of concepts, the particular path which should be 
fol lowed, the degree of difficulty of the problem to be generated, and the 
depth of monitoring and explanation of the problem solution. - ~ * . t r 

SS^r. Since there a r^e a large number of concepts ava 1 1 a b le f or study, the >".*:; 

System attempts to select the next concept In such a way as to make optima^ 

Suse of the student's time. The goal Is to pace the student through the conV 
cepts gu JcR;Ly enough so that he does not become bored or unmoti vated and yejt 

^hp-t so f ast that he Leccmes undu 1^ contused. ; : 
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i 
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REGISTER 
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JBJ CONCEPT A IS A PREREQUISITE OF CONCEPT B. 



B 



CONCEPT A IS A PREREQUISITE OF CONCEPT By 
CONCEPT A MAY BE USED AS A SUB-CONCEPT BY B. 



Note: The relation "Is a prerequisite of" Is transitive 

(A Is a prerequisite of B, B Is a prerequisite of C, 
• prerequisite of C) v-w- '"- 

W '! ^ ' ' - FIGURE 2 CONCEPT TREE 



les A Is a 
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Each student Is assigned a master average when he first logs onto the 
CAI system. Thfs could be a function of his I.Q. "cr class standing (In the 
past, each student has been arbitrarl ly assigned an intisl master average 
of 2). This value changes 3S the system gains experience witl^^ student. 

A student ? s master average controls the speed with which he Jumps 
from one plateau of the^concept tree to the next* "In order to jump^to .the 
next higher plateau, the average of his levels of achievements in aU concepts 
at and bellow the current plateau must exceed his master average. Consequently, 
the lower a student's master average, the faster he wf II progress% Each 
student ? s master average is updated after the ccmpl etfon of a, problem . _ 

Once the student f s plateau has been determined, the system selects a 
set of candidate concepis from this plateau and those below it If necessary. 
In order to qualify as a candidate concept, the average of the student ! s 
: levels of achievement in all prerequisites for this concept (as determined 
for the concept tree) must exceed his master average. If this Is not the 
case, the prerequisite concept In which the student has the lowest level 
Is selected as a candidate in Its place* This provides for automatic review 
of selected concepts at I owe plateaus. ;i; 

The system then chooses one concept from among the candidates. Each 
concept is evaluated based on a number of factors such as the time elapsed 
since its last use, the "stability 0 of its current level , the sign and magnitude 
of its most recent level change (negative changes are weighted more heavl ly) , 
and Its relevance to other concepts as determined by the number of branches 
of the tree connected to It. Each of these factors Is multlpl led by a weight. 



1 w 
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The highest scoring concept Is selected for presentation to the student." 
The student always has the option of vetoing this selection and choosing 
his own concept or accepting the system's second" best choice. 

If the course- author Is not satisfied wi^ the^nianner in which the 
system Is selecting concepts^ It Is relatively easy to modify the concept 
tree or change the weights associated with each of^he factors affecting _■ 



-concept^ selection. The concept selector 5 s completely sou rse- Independent./ 
^Consequently, It can be used In conjunct I on with another course If It Is 
provided with a new concept tree. - 

It Is also quite easy to add new concepts to a course. A solution 



algorithm and problem generator must be programmed for each new concept. A 
c~ ;4 I Ing sequence Is established which ensures that the problem generator 
passes the solution algorithm all necessary problem parameters. This 
sequence Is appended to the Ms1 of concept calling sequences already " 
available. The next step Is to enter the author-mode of the generative 
CAI system In which the course-author or Instructor Is assisted In making 
the required additions to the concept-tree. The Instructor can also examine 
student performance records while In the author-mode and see how his class 
Is progressing. vryV^^ "V : i-V -- 1-\ -> ''. _-"_f_ -~ 55 

v " Table 2 gives an example of the Interaction between COMSEQ and a student. 
The f I rst letter of each student response Is under I Ined. The student is attempt- 
ing to determine the combinational logic required to build a sequential circuit 
for the state table shown. ^ 



I 
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. TABLE 2 
Sample Student-System Interaction 

Derive the truth table for the sequential network 
which yields the following state table. 

I current |inpu?-xi I 



STATE 


|xi=o'xi=i| 


QO 


|Q2/0 Q1/0| 




Iqi/o qo/o| 


Q2 


|Q2/0 qo/o| 


Q3 


lQO/0 Q2/0| 




(NEXT STATE 



/output! 



Use SR 



Is this problem ok? 



How many Flip-Flops are needed? 

No. Since, there are 4 states, 2 Flip-Flops are needed. -V ' 

3 Flip-Flops would be used for 5 to 8 states. ' 

The following is the state assignment: 

Each state will assume its normal binary value 

STATE Y1Y2 : ■-' 

QO 00 \ 

Q 1 01 *y r -<', A --^z^--^y .. 

Q 2 10 ! 

Q 3 11 . - ^ i : * f.t- 

The table for describing the required combinational logic follows: 
| | | CURRENT i HEXT j | - | | \ | :" : 

| | INPUT | STATE | STATE | FF1 | FF2 | | OUTPUT | 
|ROW| fc. |Y1Y2 | Y1Y2 | SI Rl| S2 R2 | | | 

I o| o|oo I :}"] I I / |. ( I 

What is the next state for row 0? 

5.1 1 : v. _ - : - \ ■ . 

Ko . The current state corresponds to QO. 



I 
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TABLE- 2 continued 



The next state for input XI « 0 is Q2 
This corresponds to Y1Y2 * 10 
What is the value of the output? 



Type in S3 



Type in Rl 



Type in S2 



Type in R2 



Your answer is ok but a better answer is S2 = 0, R2 « B (D is a f don ? t care?) 



|R0W| >X1 |Y1Y2 
I Of 0 | 00 

i ll o I ci 



|Y1Y2 
I 10 



SI Rl S2 :R2: 



OUTPUT 



1 0 



What is the next state for row 1 ? 



Note: The student contl nues bu I Id T ng the truth table for SI, Rl, S2 , R2. 
After he finishes, he can continue with CAl and minimize the combinational 
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C. Problem Generation and Solution 

As can be seen frori the concept tree, there are nineteen distinct concepts 
covered by COMSEQ. Associated with each concept Is a problem generator and a 
solution algorithm. The prcM em generation routines are very much Interrelated 
1 n that they share several cannon subroutines. The subroutines are of two" 
basic types. The first type is driven by a probabilistic decision taWe. A 1 1 
pflthe decisions which affect problem complexity are Identified and comprise 
the rows of the table. There are nine columns as there are at most, three 
possible outcomes for each decision In each of the three LEVEL ranges (0-1, 
1-2, 2-3). In Table 3, the entry In row 2 column 8 means that the probability 
of a state-tab ie problem having one input bit is .2 for a student with LEVEL bet- 



ween 2 and 3. y \ V _-„-\- : ; _ ; _ > \ __/-_ ' 

/ The second type of problem generator is driven by a probabili stic grammar. 



A probabilistic grammar is a formal language in which each rewrite rule is 
assigned a probability cf being applied. For several problem types, the 
generation process is controlled by both decision tables and probabilistic 
grammars. . V / - -=.-." .~:~Sr -~- _ ; / - -."*-: _ \M 

W ; ; Table 4 Is an example of a probabilistic grammar which generates - 
logical expressions for use in the concepts deal Ing with truth tables and ^ 
gpguential circuit analysis. Each of the symbols to the left of the arrow? Y r 
f^js a non-terminal symbol and can be replaced by the string of symbols to the 
Sfetgh+^ the arrow. The incomplete logical expression Is scanned from \ett^ 
^^r^lght /for hpnrter^ When the non-terminal syrnboT A Is^: 

g&^dV^ Increasing the length of the; 

SxpressToh where: . - -; 

PJ-f) «, P- K z(-t)l= .^-5G/-( n( t) * 1 .50 1 .5) ^ (I) 
a d - 



i 

i 
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TABLE 3 

Decision Table for Generating State "able Problems 



llumber of states(2|4|5-8) 
Number of input bits <0 j 1 1 2) 
Output (absent | presentj-) . 



Level Range 




2-1. 



.5 « .5 j 0 j 0 



.5 .5 j 1 .5 .2 i -8 I 
.5 1 - ' ! .2 ' .8 | - i 



8 



I 

is 



I -i 



I 



TABLE 4 
Grammar for Logical 



fens 



Probability: Rewrite rule 



P i . 


MA* 


: a 




V • 


Mn 


P 

cl 


A->P 


p - 
c2* 


A->Q 


p v. 


A-»R 


P 

c4 


A+S 



P d « *->•*/ (or) 



P d J **A< 



P g : *^(K/ilH>) 



e 



P e i *->(+) (exclusive or) 



Constraints 0 <_ ? a , P fe , P cl> p P < 1 

" : ; , "a + P b ♦ I t P d ■ 1 

' - • 2P, + 3P » 1 

a _- e - 



j 
I 
I 
I 
I 
I 
I 
I 
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where n(t) Is the current length and C Is the number of variables In rhe 
expression (C >_ 2). 

Sl.nce (P a -+ P b > Is Inversely proportional to the current length; the , - 
logical expressions do not become unwieldy. If the random number generated 
Indicates that the expression should not be extended, one of the terminal 
symbols {P, Q, R, S} replaces A. 

If the non-terminal symbol Is * , one of the five logical operators 
{V, A, + , + , 0} replaces It. P Increases with LEVEL, while P. decreases. 

- - B - - = _ O r 

Hence,, the more difficult operators are more I.Ike I y In expressions generated 
for advanced students. 
|;: ^ ; v Once the problem has been generated, all pertinent parameters are passed 
to the solution algorithm. As each sub-task of the solution Is completed, a 
decision Is made whether or net to question the student on Ihis part of the 
solution. Students with LEVEL < I In the concept will, of course, be asked 
the most questions. The student's LEVEL and, hence, the depth of monitoring 
Is dynamically changed as the problem continues. Each correct answer causes 
an Increase In LEVEL (.07 <|AL|<.28) while each Incorrect answer causes a 

decreases In LEVEL. " v.-fe- •, 

0. Error Analysis ; ] \- * .-r /. " - -_. \ : 

When a student Is Incorrect, the system analyzes his response to deter- 
mine the cause of his error and provides pertlment remedial feedback. ' The 
concept which teaches the use of Karnaugh Maps Is a good example of what 
can be done In the way of error analysis. In the sample problem shown In 
Tab ' e 5 , the student Is attempting to Identify all squares with truth value 
I wh Ich must be picked up In pal rs ("2-SQUARES") . The only requ I red pair - 
consists of m In terms 8,9. The output shows the remedial response given to a 
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. TABLE 5 
ANSWER ANSLYS IS IN KARNAUGH MAPS 

PROBLEM : Minimize a function which has 5, 7, 8, 9, 13, 15 as minterms using 
the Karnaugh Map method.* 
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^20^ ^ KARNAUGH MAP 
Part of Student Solution 



PLEASE ENTER THE 2-SQUARES. :\\ . : A 
WHEN YOU ARE DONE, TYPE 99. 

4 'DOES NOT HAVE TRUTH VALUE 1. * 
5 , 7 

THESE TERMS ARE PART OF A 4-SQUARE. (A) 
5, 15 : ^ ^ : : -~ r- 

THESE TERMS ARE NOT ADJACENT (B) 

13*,9 IS A 2-SQUARE, BUT IT SHOULD NOT (C) 

BE PICKED UP YET. 

8,9 

CORRECT. ENTER MORE 2-SQUARES OR 99 . 
9? 

PLEASE ENTER THE 4-SQUARES. 
5, 13, 7, 15 

CORRECT. ENTER MORE 4-SQUARES OR 99 

99 ^-^Yrjy^KK ■- ~' : v 

THE FINAL SOLUTION FOLLOWS: 
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varlety of student errors. It Is produced by calling the parts of the 
solution algorithm which check for 2-SQUARES and 4-SQUARES and analyzing 
the results. 

_ T Response A Is provided since mlnterms 5 and 7, as well as forming a 
valid 2-SQUARE, are also part of a 4-SQUARE. This group of 4 (5, 7, 13, 15) 
Is, of course, preferred. Response B results since no valid 2-SQUARE exists 
which contains mlnterms 5 and 15. Even though 9 and 13 form a valid 2-SQUARE, 
each of these mlnterms Is also found in another 2 or 4-SQUARE. Hence, the 
algorithm correctly decides to defer selection of this pair as indicated by 
response C. , -„- v • ' = --- , 

1/ : - It Is Important to note that th I s type of deta 1 1 ed remedial feedback 
Is automatically avail ab |e regardless of the 3-var table or 4-variable 
Karnaugh Map problem being attempted and for any student error. '. _ _ 
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ill. The MALT System 
A. Overview 

The subject matter of MALT- Is somewhat different from COMSEQ and . - 
GAILD in that It teaches machine language programming. However, MALT 
does utilize several of the concepts from COMSEQ In the problem solver 
.mode to simulate the execution of a studentj program. These Include the.. - 
concepts dealing with octal arithmetic and logical operations. _ - 
=_..-_ In machine language programming, there are certain basic concepts which 
-must be used over and over again In the design of a complete program. For 
example, pointers to data must be Initial Ized, counters (to keep track of 
the number of loop Iterations) must be initialized, masks must be set up, 
program loops must be terminated, data must be transferred Into and out of 
memory, and accumulator and overflow status must be checked . - :-_--=- - ; 



There are also somewhat more specialized concepts which often use these 



basic concepts as subroutines. For example, the concept which prints out 

the contents of a register uses the basic concepts concerned with transferring 

data Into memory. Thirty-five concepts have been I so I ated as essent I a| 

modul es of machl ne language programs. As such, they can be comb Ined to form r 

a wide variety of rather lengthy and complex programs. This is the key 

factor which enabled the design of the MALT system. - 

The concept routines interact with the student during the design of his 
solution program. They sol Ic It program statements from him and present 
remedial feedback if his response is Incorrect. Figure 3 Is a block diagram 
of the MALT system which shows the Internal and external (STUDENT) flow of 
information. - 
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The previous student performance determines what type of problem will be 
generated. This problem is presented to the student In natural language 
and also passed on to the system as a concept sequence and parameter list. 
Next, a list of logical sub tasks for each part of the problem is presented 
to the student. The system representation of this ,: f low-chart" Is a 
sequence of calls to concept routines. These concept routines solve the 
programming problem and interact with the student to monitor his solution. 

As the student undertakes each sub- task In the programming process, a 
corresponding Concept Routine Is entered by the CAI system, which guides 
the student through the construction of 'that part of his program. During 
this phase, the student Is constantly being given feedback as to the correctness 
of his program. If his program Introduces logical errors, the system will 
point these out and offer helpful suggestions for their correction. If the 
system feels that the student might benefit from observing his program In, 
operation, It also has the capability to simulate statement by statement 
program execution. - ^ r - V : -"- - - * ' fP-= - 

The system Is constantly evaluating the student's performance and updating 
his permanent f I le. This Is necessary because his achievement determines not 
only the difficulty of the problems given him, but also the amount of Interaction 
which he receives during the design of his program. 

: Table 6 provides a complete example of the operation of MALT. The 
comments In square brackets have been added to clarify this example for 
anyone not f ami liar with the PDP-8 instruction set. Each of the 400g memory 
■registers is 12 bits In length. The first octal digit of an Instruction is 
the operation code. The last three octal dlgltf specify^ the address of the 
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MMT In Operation 
TABLE 6 

Your problem is to write a program uhich will: 

Read in 20 (octal) ASCII characters and store them in registers 232 through 252. 
Form the sura of registers 232 thru 252 in the accumulator • 
If this results in a non-zero LINK, stop with the (ACC)=7777, otherwise stop 
with (ACC)=0000, 



Is this problem OK? 
_yes - 

Here are the sub- tasks for the 1st line. 

1) initialize a ptr to register 232 

2) initialize a ctr with the value of -20(octal). 

3) Read in a character. 

4) Store it away using the ptr. 

5) Updats the ptr. 

6) Update the ctr and if it is not zero, jump back to start of loop. 
SUB-TASK i: _ s ? 

Since we may need a ptr later , we will store the ptr value in -memory 
and move it to a register which we will usa as the actual ptr . 
Use M377 to store the ptr value, and M376 as the ptr. 
M377: ■ " -\ . 

J0232 /; V ; ; ; V -'■ _ r - . " 

Now, get the ptr value: 

11000: - : - ._ 

JL376 [Add in contents of M376] ~_ 
WRONG! - "= . 

lf TAD 377" is coded as :1377 v ' -v . 
And deposit it into the ptr: - _ 

M001: : 
3376 [deposit ACC into M376, clear A< 



'SJWTASK 2t 

fe (will store the ctr value in memory and then move it into another 
register for use as the ctr. ~* 

Use register 375 to store the ctr value and register 374 as the ctr. 

M375: - ^ ^ - 

J7757 . 

7757 can f t be used as the ctr value. Try again! 

M375: '-_ _ 

J7760 [-20 in 2 f s complement form] 

First, get the ctr value: 

M002: - 1 ^ -: 

J.375 

And deposit it into the ctr. 
M003: 
3374 
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SUB-TASK 3: 

First, check the reader flag. 

MOOt: 

__5031 

NO! 

To check the reader flag is 6031 

Next, jump back 1 register is it's not ready. 

M005: 

_5004 [JUilP to MOW] 
Now, operate the reader. 
M006: 
6036 



n 



SUB-TASK 4: 
M007: 
J3376 = . 
Wrong! 

UDCA I PTR" is : 3776 [This should be an indirect deposit] 



n 



SUB-TASK 5: 

Q^, ^update the ptr. 

M010: - >. 

_2376 -. > ; _. - \"" \ 

SUB-TASK 6: . 

Now, increment the ctr. 

MOU: " r =--^- - J 

_2374 

And if not zero, jump back to start of loop. 
H012: ' 
J5002 ~~ -■ ; V 



The loop starts in register 004- so it is :5004 - 

Here are the sub-tasks for the 2nd line. * - = 

1) Initialize a ptr to register 232 

2) Initialize a ctr with the value of -20(octal) 

3) Add in the next register in the list. 

4) Update the ctr and jump back if the ctr is not zero. 

SUB-TASK 1: 

Register 377 already contains the ptr value, 0232, so use it with M376 as the 

ptr. ; 

Now, get the ptr value: 

M013: 

J.377 

ASd deposit it into the ptr: 

M014: 

_3376 

Sub-Task 2: 

We can re-use 11375 as the ctr value and H374 as the ctr. 

M375: 7760 
First, get the ctr value: 
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operand. If this is a number > 400 g , the instruction is an indirect address 
instruction. In this example, Indirect addressing is used in conjunction 
with a pointer to enable an operation to be performed on a group of adjacent 
registers. -A counter Is used to control the number cf times the operation 
is performed. 

M377 stands for the memory register 377. The student's program starts 
at M000. The first character of each student response is underlined. The 
dialogue shown is that which would be received by a beginning student. 
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B * Problem and Logic Generation 

Each programming problem can bo thought of as consisting cf three 
distinct phases; an Input phase, a processing phase, and an output phase. 
There Is a set of problem primitives associated with each. A problem 
primitive Is a parameterized statement. Jp = { I U P UO} Is the set of 
problem primitives where I,P,0 represent the sot of Input, processing; 
and output primitives respectively and Is their union. The nul l primitive 
Is also an element of l,P,0; consequently, many problems will have fewer 
than three phases. 

There Is a function d defined oyer the real numbers f rem 0 to 3.0 isuch 
that d(e) Is the subset of Jp which may be used In problems for a student 
whose LEVEL is e. In addition there is a function f defined over { I U P} 
such that f(l } ) Is the subset of {P} which may follow Input statement l { In 
a meaningful problem and f(P J is the corresponding subset of {0} for P.. 
The purpose of these functions Is to insure that the difficulty of the program- 
ming problem generated Is suited to the student f s ability and that the problem 
K makes sense". * ~~ "- - ■* -* \ - = ~ ^ ; 

Problem generation, then, proceeds by f I-st applying d to 0 . An input 
statement, I j , Is selected at random from the set of eligible Input" primitives. 
Evaluating f(lj) further reduces the set of eligible processing primitives. 
One of these, P^ Is selected at random and this process Is repeated to select 
0 K * A final check is made to determine that the problem Is sufficiently 
different from those already worked by this student and then the values of 
parameters In the selected primitives are , randomly generated. Table 7 
gives an example cf some resulting problems In order of Increasing difficulty. 
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TABLE 7 
Sample Problems 

Note: All randomly generated parameters are underlined 
Level Range 0-1 

1. Add the contents of register J50 to the contents of register 167." 

2. Print out the message " HELLO" . 

3. Read In a series of ASC-II characters ending with a * and store 
them starting In location 120 

L ; 4. Reading ASC-M characters and store them starting at location " 
30(h Search regl ster 300 through 330 for the largest number . 
Level Range 1-2 

^ I. Read In a series of 3-digit numbers and store them starting at 
jpr . * location 250. The input will end when the first character of a 
S^fr number Is a "X". " -:L - - - : ; ; ; 

2 ' Read f n 21 (octal) four digit numbers and store them start I ng at 
W V location 242. Search registers 242 thru 265 for the 1st number 

which begins with the octal digits "70" . (example 70XX) = 
3 * Mu '+'My +he contents of register 2M_ by the contents of register 
IK '" 3 J°L ' * • ' 

Final ly print out the 4-digIt contents of the Accumulator. 
'LeveT Range 2-3 * J 

^ I. Search registers J60 thru 205 for the octal number 7215 . 

For registers J60 thru 205, print out the register number, 4 spaces, 
and the octal contents of that register. 
2. Assume a table has been set up starting at location t20 consisting: 
of a ^character symbol followed by a number; there are 1£ of these 
entries. 

Search the table for the symbol 'Ml* and- retrieve the correspond! no 
number, if it Is not In the table, then halt the program. 
Finally, print out the 4-digit contents of the Accumulator. 



-26- 



in 
I n 



Associated with each problem primitive is a string of two digit numbers 
called the concept sequence. This sequence indicates which of the thirty- 
five basic concepts cf machine* language programming must be performed and in 
whet order to program each primitive correctly. TfieLaverage number of concepts 
needed per primitive is six • 

Each primitive Is programmed separately ; hence, the concatenation^ 
the concept sequences for each of the three selected primitives Is the " 
concept sequence for the complete problem. This sequence specifies the iorm 
which the student's solution must fellow and Is equivalent to a flow-chart. 
It is interpreted by the LOGIC GENERATOR and printed as a list of sub-tasks 
prior to programming each primitive. .Table 8 shows- the concept sequence 
for the problem being programmed in Table 6 . ~ \ 

f This restriction on the form of the student f s program is essential in 
X_ qrOer for MALT to verify the correctness of thesfudent 1 s program and help 
him with the coding. Since the student Is a neophyte In machine- language 
programming, It is felt that this imposed structure is useful In showing him 
j how to attack a programming problem and outline Its scl ution . He does have 
considerable freedom In coding each sub-task as wl 1 1 be discussed In the next 
section. , -.. - 

C. Program Coding and Verification 

The coding of each sub-task Is monitored by the concept routine In the * 
: sequence responsible for that sub-task. If the student's LEVEL exceeds the 
generation threshold for the routine, the coding for the associated sub-tasks 
(or sequence of sub-tasks) is provided by MALT. If the student -s LEVEL Is 
very low, he Is led by the hand and program statements a~e requested one at a 
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TABLE 8 
Concept Sequence 
Concept Sequence : 23240715103 232410 18 
The "I" following concept 15 indicates this subtask should be 
performed indirectly. 

The spaces indicate the end of a problem primitive 



Concept Routine 
03 
07 
10 
15 
18 
23 
24 



Sub-Task 
Terminate a loop 
Input ASC-II Characters 
Add a series qf ; ad j :^4nt_ ;tegisfeersi ^ 
nStbre; the^ accumulator in ry 

Check link status; : - y 

Initialise point ers - ^ 
Iriitialize counters : f 
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time. The intermediate student will normally enter a group of program state- 
ments at once. 

Several conventions have been established to facilitate the generation 
of program segments and monitoring of student programs. All user programs 
begin in location 000 and at I program const rants are placed at the top of 
memory beginning with location 377 and proceeding downwards. The middle 
areas of memory, locations 120 through 350, are reserved for lists and tables 
to be used by the student's program. 

The existence of a program loop Is assumed by the system whenever a 
pointer or counter is initialized. The physical start of the loop is the 
first memory register after the initialization process. By monitoring the 
beginning of a I oop In this manner, : the system can easi ly determ Ine if the 
student has correctly designed his end-of- loop decision sequence. The 
most common programming mistake of this kind occurs when the" student attempts 
; to jump back to the initialization sequence instead of the ma in body of the - 
loop. . =.. 1 v . '- ^ - . .. * : 

Another P r °9 ram parameter which must be kept track of is the accumulator 
status r The simulated computer has neither a non-destructive deposit nor a 
destructive load instruction. Hence, the accumulator must be cleared orior 
to loading It with a given number and must be reloaded after a number has 
been deposited in memory, if the number is still needed. The status of the 
accumulator is going to determine which of several alternatives is to be 
pursued by certain of the concept routines in the design and checking of a 
program segment. 

A complicating factor in determining accumulator status is the existence 
of logical branching or program jumps. The accumulator status may differ 
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depenalng on whether a concept routine was entered sequentially or through 
a program jump. 

Forward jumps to yet unprogrammed concept routines also present a problem, 
as the memory location In which the new routine starts is net yet known. 
Consequently, MALT keeps track of the first memory location of each concept 
routine. ' If a jump is made to a' previously programmed routine, the required 
Instruction Is provided Immediately. In the case of forward jumps, a note 
Is made of the memory location In which the jump Instruction belongs and 
the concept routine to be reached. When this concept routine is finally 
programmed, MALT completes all prior Jump instructions -which reference It. 

There are two important techniques used by the MALT system to judge 
the correctness of a student's program. The most common method is to analyze 
in detail each segment of the program as it is typed In, to determine if it 
performs the required functions. This is done on an instruct ion-by- instruction 
basis so that there is immediate feedback to the student. 

;- Immediate verification Implies that the system must have a detailed 
knowledge of the status of the user's program at al I times. As the student 
formulates each response, the system also generates what it considers to be 
an appropriate answe r. If the two do not match, the system must determine 
If other responses are possible. If so, the student's answer is compared 
with al I such reasonab le poislbfl ities. When the system final ly decides - 
that the response supplied by the student Is In error, It Informs him as to 
the reason for this determination and supplies the best program alternative. 

If the student's response matches any o.f those which the system generated , 
then It Is accepted by- the system as a valid alternative to its own solution. 
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Since this was not the expected result, however, the system must adjust its 
representation of the users program status to reflect the new conditions. 

In the rare event that there are too many acceptable ways to program a 
particular sub-task, the program segment supplied by the student is simulated 
to determine Its correctness. To verify the user's program through simulation, 
all conditions of the machine which might possibly affect final program ret- Its 
are determined. For example, If the. program is intsnded to perform a particular 
operation depending upon the status of the overflow Link register, then only 
two initial states are necessary; the program is tested with a zero Link 
and again later with a non-zero lil<nk. 

Once the various initial states have been determined, the program 
segment can be simulated under each condition. The system decides, following 
each simulation, if normal program termination occurred. Cond itions which 
m ight- cause abnormal termination are such things as infinite loops, undefi ned 
instructions, or program branches which are directed outsi de the user's 
program segment. Any such conditions are corrected immediately by the student, 
the current set of initial conditions is re-established, and simulation is 
attempted again. " 

,f an y particular terminal condition Indicates that the user's program 
did not perform its function correctly, MALT attempts remedial action. Since 
it is aware of the exact results which should have been obtained, it can provide 
a concise description of the error. It cannot, however, isolate the location 
of the error in the user's program. This determination must be left up to 
the student. However, the problem nas been greatly simplified due to the 
system's diagnostics and the user's ability to observe his program in execution. 

If the student is unable to correct his program segment, MALT will generate 

a correct program segment for him. 
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IV. THE CAiLD SYSTEM 
At Overview 

The final system to be described Is concerned with teaching students 
how to use Integrated circuits (I.C. f s) to realize some of - the "paper designs 11 
problems encountered In COMSEQ. As many of the students enrolled In this 
course have had nc prior exposure to digital or even analog electronics, 
CAILD performs an essential service. Figure 4 fsa block diagram of CAILD. 

The student first constructs his circuit off-line on a special student 
Logic Board. He has a fixed set of I.C. f s to work with which includes 
/several types of NAND gatesy Inverters", four J K Fl ip-F I ops, aod four D ■ 
FI ip-FIops. The output of each gate can be monitored directly by CAILD. ' 
There are too many gate j npu + points to justify giving CAILD the abi Ilty to 
monitor them all. Consequently, a test probe is provided which is connected 
to CAILD through an A/D converter. The student can move this probe to any 
/test point in the circuit. - _ = : 

Once, the student has constructed his circuit, the logic board Is .. 
connected to the general-purpose Input/Output buffer of the PDP-9 computer. 
The student describes his circuit by specifying each output equation and, 
in the case of sequential circuits, each FIIp-Flop control equat* n in sum- 
of-product form. As each equation is entered, It Is checked for syntax 
errors and ail variables used are noted. Eactf equation is then converted 
to NAND format and stored in memory. 

Debugging proceeds by testing the output cf each equation for all 
possible combinations of the fnput and state variables. CAILD sets the input 



I i 



I 
J 



9 

:RLC| 



i 
I 
1 
I 
I 
I 

! 



I 



1 
I 
I 
1 



■ -33- 

and state variables In the circuit to correspond to the combination being 
tested. The logic value present at the circuit point under test Is compared 
to the value predicted by the equations. If they differ, CAILD aids the 
student In tracing down the circuit error and wajts for him to correct If. 
CAILD then retests the equation. 

When all equations appear error free, the circuit Is free of any wiring 
errors or faulty components. In order to verity that the circuit actually 
does what the student Intended, CAILD allows him to test It out. The student 
specifies a set af test conditions which are applied by CAILD to his circuit. 
The resu It Ing output val ues and , In the case of sequent la I c I rcu Its, next- 
state values are displayed to the student. * • \ ~~ ; M 
rvV CAILD can be used to test and debug a variety of digital circuits. 
As has been mentioned, there are two Input, two output, and four state 
variables for sequential circuits of up to sixteen states. Combinational 
circuits with up to six inputs and sixteen output equat ions can also be 



designed. The additional input lines are obtained by allowing the F Hp-Flop 
state variables to serve as input variables. *^ 
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B. Error Detection 

There are -three categories of errors which may be detected. 

1. Wlrinc, error at input of I.C. 

2. Improperly powered I.C. 

3. Faulty I.C. 

The presence of an error Is Indicated by a disagreement between the 
monitored logic value at a gate output poln. and the simulated or calculated 
value. To trace the error to Its source, CAILD checks the logic value at the 
output of each tower level gate which Is connected to the Input of the gate 
In question. If any lower-level gate output Is Incorrect, then CAILD will 
switch Its attention to this gate and Its Input points. 

If all of the lower level outputs are correct, then there may be a 
wiring error. This Is determined by monitoring the logic values at the =- 
corresponding input points of the gate In question. If there Is a disagreement, 
then a wiring error exists. If there Is no djsagreement, then the gate In 
question Is either faulty or powered Incorrectly so CAILD will next check 
the voltage values at the power and ground points of this J.C. Figure 5 - 
Is a flow chart* of the complete test procedure. 

- !n all of this, the student is actively Involved. He musi refer carefully 
to hK wiring diagram and Inform CAILD of the location of all relevant gate " 
output points. He must also physically move the test probe to all gate 
Inputs which are to be checked and all suspicious I.C. power and ground points. 
Table 9 shows the Interaction between CAILD and a student during the debugging 
process for the wiring diagram in Figure 6. The first error found is the 
absence of a ground connection to the inverter. The second error is a missing 
wire between M57 and M50. All student responses are followed by a •';». 
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Figure 5 Flowchart for Debugging 
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TABLE 9 
CAILD In Operation 

EQUATION UNDER TEST: 
Zl=(Xim)-K-Xl+Y2 )-KYl+Y2 ) ; 

DO YOU WISH TO PRINT OUT ALL TEST CONDITIONS ? 

(Y or N) 

M; 

SIMULATED OUTPUT CONDITION: 0 
ACTUAL CIRCUIT OUTPUT CONDITION: 1 

INPUT CONDITIONS: XI Yl -XI Y2 Yl 

10 0 10 

TELL ME THE OUTPUT CIRCUIT POINT FOR 

THE GATE WHOSE OUTPUT IS Zl 

p 3I; " 

TELL ME THE OUTPUT CIRCUIT POINT FOR 

THE FOLLOWING TERM: 

(Xl+Yl) 

M49; 

.SIMULATED OUTPUT CONDITION: - 1 
ACTUAL CIRCUIT OUTPUT CONDITION: 1 

PUT THE TEST PROBE ON = THE INPUT OF NEXT HIGHER LEVEL GATE 
WHICH IS CONNECTED TO THE OUTPUT OF (Xl+Yl) 
TYPE ALT MODE WHEN YOU ARE READY. 

THE WIRING SEEMS OK. 

TELL ME* THE OUTPUT CIRCUIT POINT FOR 

THE FOLLOWING TERM: 

(-X1+Y2) 

M52; 

SIMULATED OUTPUT CONDITION : 1 
ACTUAL CIRCUIT OUTPUT CONDITION: 0 . 

PRESENTLY THE SYSTEM IS CHECKING ON THE VARIABLE -XI 
TELL ME' THE OUTPUT CIRCUIT POINT FOR THE INVERTER. 
M57; 

SIMULATED OUTPUT CONDITIONS: 0 
ACTUAL CIRCUIT OUTPUT CONDITION: 1 

PRESENTLY THE SYSTEM IS CHECKING ON THE VARIABLE XI 
SIMULATED OUTPUT CONDITION: 1 
ACTUAL CIRCUIT OUTPUT CONDITION: 1 
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Table 9 (continued) 

PUT THE TEST PROBE ON THE INPUT OF NEXT HIGHER LEVEL GATE 
WHICH IS CONNECTED TO THE OUTPUT WE JUST CHECKED 
TYPE ALT MODE WHEN YOU ARE READY, 

THE WIRING SEEMS OK. 

PUT THE TEST PROBE ON THE GROUND LEAD OF THE IC. 
FOR-JK.F/F (7476).' THE GROUND IS AT PIN 13. 
FOR ALL OTHER ICS, THE GROUND IS AT PIN 7. 
TYPE ALT MODE WHEN YOU ARE READY. 

YOU HAVE EITHER AN OPEN CIRCUIT ON THE GROUND LEAD CNote: MISSING GROUND 
OR A WIRING ERROR TO THE GROUND LEAD. CONNECTION DETECTED] 

CORRECT THE FAULT. AFTER THE CORRECTION , * TYPE ALT MODE 
TO RETURN TO THE DEBUGGING PHASE. - 
TYPE ALT MODE WHEN YOU ARE READY. 

EQUATION UNDER TEST: 

Zl=(XlfYl)t(-Xl+Y2)t(YltY2); - 

DO YOU WISH TO PRINT OUT ALL TEST CONDITIONS ? 

(Y OR N) 

N; 

SIMULATED OUTPUT CONDITION: . 0 
ACTUAL CIRCUIT OUTPUT CONDITIONS: 1 

INPUT CONDITIONS: ' XI Yl -XI Y2 Yl Y2 

1 0 0 10 1 

TELL ME THE OUTPUT CIRCUIT POINT FOR 

THE GATE WHOSE OUTPUT IS Zl 

P31; 

TELL ME THE OUTPUT CIRCUIT POINT FOR " 
THE 'FOLLOWING TERM: 

(«xi+yi) 

029; . ■ - - 

THE CIRCUIT PT LABEL YOU PUT IN IS NOT A MONITORINT PT .. 

ON THE CIRCUIT BOARD.' TRY AGAIN. 

Wf9; 

SIMULATED OUTPUT CONDITION: 1 
ACTUAL CIRCUIT OUTPUT CONDITION: 1 
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Table 9 (continued) 

PUT THE TEST PROBE ON THE INPUT OF NEXT HIGHER LEVEL GATE 
WHICH IS CONNECTED TO THE OUTPUT OF (X1*Y1) 
TYPE ALT MODE WHEN YOU ARE READY. 

THE WIRING SEEM OK. 

TELL ME THE OUTPUT CIRCUIT POINT FOR 

THE FOLLOWING TERM: 

(-Xl-tY2) 

M52 ; 

SIMULATED OUTPUT CONDITION: 1 
ACTUAL CIRCUIT OUTPUT CONDITION: 0 

PRESENTLY THE SYSTEM IS CHECKING ON THE VARIABLE -XI 
TELL ME THE OUTPUT CIRCUIT POINT FOR THE INVERTER. 

-M57; . 

SIMULATED OUTPUTCONDITION: 0 
ACTUAL CIRCUIT OUTPUT CONDITION: 0 

PUT THE TEST PROBE ON THE INPUT OF NEXT HIGHER LEVEL GATE 
WJHCH' IS CONNECTED TO THE OUTPUT WE JUST CHECKED 
TYPE ALT MODE WHEN YOU ARE* READY. 

EITHER YOU HAVE A WIRING ERROR OR AN OPEN CIRCUIT L"Note: MISSING WIRE FROM 
AT THE TEST POINT. M57 to M50D 

'CORRECT THE FAULT. AFTER THE CORRECTION^ TYPE ALT MODE 
TO RETURN TO THE DEBUGGING PHASE. 
TYPE ALT MODE WHEN YOU ARE READY. 

EQUATION UNDER TEST: 
Zl=(Xl+Yl)t(-Xl+Y2)+(Yl+Y2); * 
DO YOU .WISH TO PRINT OUT ALL TEST CONDITIONS *» 
(Y OR N) - 

N; - 

DEBUGGING COMPLETED FOR ONE EQUATION. 

DEBUGGING PHASE COMPLETED.' - r M + lin rnmn / - 

DO YOU WISH TO TEST YOUR CIRCUIT? (Y OR N) LN ° +6: R^/yN^g] 

OPERATION SUCCEEDED!!! 
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V. Evaluation and Conclusions 

The purpose of these CAI systems Is to augment classroom Instruction 
rather than replace It. None of them Is really Intended to be an end In 
Itself, but their purpose Is to help the student bridge the gap between 
theory and Independent application'. Experience has shown that these CAI 
systems are far more effective than conventional homework assignments for 
this purpose. "~ 

After using MALT for approximately two weeks, students are required to 
program and run a sizeable problem in assembler language. Students who have 
acquired a feet for machine language programing through Interaction with 
MALT find it easier to strike out on their own and complete their assembler 
language program. CAILD has not yet been evaluated; however, the authors 
anticipate that the experience gained by students in debugging digital 
circuits under the guidance of the CAILD system will prove beneficial to 
jttiem in later .laboratory projects. 

Since each of these systems Is "intelligent 15 in its small domain of 
application, they are very flexible and are not limited In the problems they 
can solve. For example, CCMSEQ Is available as a problem-solv I ng assistant 
to help advanced students design and simplify the logic circuits to be debugged 
and tested by CAILD. ~ In fact, the student can use COMSEQ to derive his circuit 
equations in minimal Sum-of -Product form. 

CAILD Itself can be used to build and debug digital circuits such as 
counters, adders, decoders, and shift registers. The only constraint is that 
the number of independent circuit variables not exceed six (two Inputs and 
four state variables or six input variables)* Combinational circuits can have 
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up to sixteen output lines; sequential circuits can have up to sixteen 
states. 

Present plans call for adding a "front end" to CAILD which will 
allow a student to enter a state table for a problem or select one of a 
variety of circuits to buMd. CAILD will then aid him In determining a 
correct set of equations for this circuit. Finally, CAILD will teach 
htm how to wire the Individual l.c.'s prior to his constructing the circuit. 

Due to Its modular construction, certain of the concept routines In 
COMSEQ are used as subroutines by more advanced concepts In COMSEQ and also 
by MALT (during program s Imu I at Ion) . For example, a student who has produced 
the state table for a counter can, If he wishes, continue on and derive a 
truth table for the Ff Ip^-Flop control I ines, and also determine the minimal 
combinational logic required. 

It Is relatively easy to add new concepts to COMSEQ. A new problem 
generator and solution routine must be written. There Is an Instructor-mode 
available which can be used to make the necessary additions to the concept 
tree. Similarly, new problem primitives can easily be added to MALT. .The 
concept sequence for this primitive must be defined in terms of the thirty- 
five basic machine language programming concepts which are available for 
use In MALT. Also, the other problem primitives It can be interfaced with 
must be identified along with the LEVEL range In which it may be used. 

COMSEQ and MALT also attempt to Individualize the problem selected for 
the student and the depth and pace of Instruction provided. COMSEQ, In 
particular, learns about the student as the course progresses and uses Its 
expanded knowledge of the student to decide how quickly 'he should progress 
from plateau to plateau In the concept tree and which one of the concepts 
within a plateau should be studied next. 



The authors feel that this set of generative CAI systems will 
continue to be useful In augmenting the learning tools available for a 
beginning student In computer science and electrical engineering. Hopefully, 
the concepts employed In developing these systems can be applied successfully 
In other course areas. Current research Includes the design of a system 
patterned after COMSEQ for teaching problem-solving in high school algebra. 
An author language for facilitating the design of future CAI systems Is 
also being Implemented. 

_ A questionnaire was distributed to students using COMSEQ and MALT 
this past semester (Spring 1973). - A summary of student- responses to some of 
the questions In thji questionnaire Is given In Table 10. These results 
clearly Indicate student acceptance of the role fulfilled by these systems 
in the course. 
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TABLE 10 

Student Evaluation 

Note: Questions 1-8 refer to MALT only; 10-17 to COMSEQ only 

For questions 1-15 the numbers of students giving the following responses 
are tabulated 

Strongly Disagree Uncertain Agree Strongly 

Disagree Agree 

1. The system was useful in introducing me to machine language programming. 

■ 0 2 1 _ 18 12 

2. It was relatively easy to learn to use the batch version of the assembler 
since I had been introduced to programming concepts through MALT. 

0 5 4 15 7 

3. Since the subtasks were always laid out for me, I felt very constrained 
using MALT. - - 

0 19 9 5 0 

k. Because .the subtasks were laid out* -I only learned the mechanics of 
programming and didn't .really understand what was going on. 

1 - 16 8 5 2 

5. The approach taken in printing cut the subtasks was good as rt taught = 
me how to organize a machine-language program. 

0 2 7 20 ■ 4 ' 

6. The problem became more difficult as my level increased. 

1 3 7 19 3. 

7. There was a good variety in the problems I received in MALT. 

1 12 6 13 = 

8. In general, I enjoyed the interaction with MALTv - ~- 

0 3 6 21 3 

9. In general, I preferred the use of CAI in this course to conventional 

homework (both CAI systems) 

0 ^2 4 11 16 

10. I was concerned that I might not be understanding the material. 
.3 13 if 7 1 



l 
I 



11. CAI is an inefficient use of the student 1 s time. 

11 12 '2-^ 2 2 

12. CAI made it possible for me to learn quickly. 

1 6 7 15 0 

13. The CAI System does a good job of selecting concepts. 

6 6 8 8 1 

14. In view of the effort I put into it, I was satisfied with what I 
learned while taking CAI. 

2,1 0 17 8 



i i 

i 



15. Overall, I enjoyed working with CAI. (CONSEQ) 

0 2 2, 23 2 

For questions 16-17 the numbers of students giving the following responses 
are tabulated" . 



All of the Most of ^ Seme of Only _ Never 

time the time the time Occasionally 



i I 

m 



16. I found myself just trying to get- through the material rather than 
trying -to learn. _ 

1 = - .5 5 13 4- 

17. I was given, answers but still did not understand the questions. 

0 . 1 13 5 - 10. 



VI. Problem Generation And Solution In High-School Algebra 
A. Overview 

This chapter will describe a generative computer-assisted Instruction 
(CAI) system for hlgh-schcol algebra. The classes of problems which the system 
can handle are word problems, manipulation problems, and graphing problems. Word 
problems test a student's reasoning and problem-solving ability; manipulation 
problems test his grasp of the fundamental skills and techniques; graphing pro- 
blems test his basic understanding of graphic methods and ability to plot 
equations. 

Problem generation for all three types Is based on a probabilistic 
grammar. A fable of probabilities covering each rule In the grammar Is used 
to tailor the problem to the Individual student^ In terms of desired emphasis 
and level of difficulty. For-example, the table nglght favor generation of 
ar "age" problem If additive concepts were- to be emphasized; whereas, a 
"percentage" problem would be more likely If multiplicative concepts were to 
be emphasized. " If a difficult problem Is desired, the probabilities associated 
with complex problem primitives or complex expressions would dominate. , 

.Like problem generation, solution monitoring Is dependent on a student's 
past history. When a student begins a new concept he will be led step by step 
through the problem and essentially "learn by example". As his proficiency 
Increases, the system will-require less and loss Interaction and provide less 
Instruction. Solution monitoring (after the student Is sufficiently advanced) 
consists of checking to see that his approach appears reasonable, accurate, and 
error free. If the student appears to be having trouble or If he asks for help, 
parts of the solution will be given* 
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B. Manipulation Problems 

Manipulation problems are used to Increase the students skill In 
algebraic operations. For example, if 4X+3=7, what does X equal? The 
Intention Is to supply a different type of manipulation problem for different 
concepts. to be emphasized. In order to obtain different types of problems 
from one generation system,. a prababl I Istic grammar Is used to generate each 
expression of the equation. See page 13 for an explanation of probabilistic 
grammars). The grammar Is implemented as a set of recurrslve PL/I functions. 
A table of probabilities Is used In the call to the generator routines to 
supply Initial probabilities for each rewrite rule. Also In the table Is 
a set of decrement values wh ich ^are used to establish a linear conditional 
probabl lltles effect.^ The currently used grammar is shown In Table 

In order to generate problems of a specific type, the appropriate prob- 
abilities table need only-be passed to- the generating routines. An Inter- 
active program Is available to the course author so that he can experiment 
with the probabilities table for each concept before Incorporating It Into 
the concept Itself. = 

As an example, If the derived problem type Is a quadratic equation In 
one variable, the probability and decrement for a following term are set to 
values which tend j*o produce three or more^terms In >he final expression. Each 
*term> Is rewritten as <factor>. The decrement values are applied between each 
call, to <term> . Hence, the probabilities associated with each re-write rule 
of <f actor> -are different for each term of the expression. That Is, the 
probabilities are adjusted such that In the first term the re-wrlte rule 
<factor>-HcVAR>+<POWER> Is chosen; In the second term, <factor>+<CONST VAR>; 
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Grammar fcr Generating Polynomials 
TABLE . 1 1 

1. EXP+TERM {+ TERM} | {-TERM} 

2. TERM-FACTOR {* FACTOR} | {/FACTOR} 

3. FACTORKEXPHN|(EXP)|VAR|CONST VAR | VARtN | CONST 

4. VAR X|Y|Z 

5. CONST-)- 0 1 j | ... 1 9 1 CONST CONST 

6. N-H|2|3 



I u 



There Is a probability associated with each re-write rule. 
KEY: Rewrite Rule I states that an expression can be. replaced by a single 
term fol lowed optional ly ^y.a' pPus or minus sign and anoiher ter^u 
(Any quantity enclosed In brackets is optional). 



I 
I 
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Word problems are used to Increase the students problem solving abilities 
In the sense of converting a verbal description of the problem Into equations. 
As in the case of manipulation problems, the word problem generated should be 
appropriate to the concepts to be emphasized. An additional difficulty 
In generating word problems Is that the solution must be generated as part of 
the problem generation procedure. For this reason, even though the word 
problem generation routines can be modeled as a probabilistic grammar, they 
are actually Implemented as a set of PL/I routines which manipulate a LISP- 
like data structure. The data structure (actually there Is a separate data 
structure for each problem type due to storage I lmltatIons)ccnta Ins a set of 
problem primitives which are strung together In order *o construct a problem, 
the data structure Imposes a difficulty ordering- on the primitives, associates 
a solution primitive with each problem P r Imltlve and addition* iy Indicates 
which primitive can reasonably follow from another In order to make a meaning- 
ful problem. As with manipulation problems therejs a table of probabilities 
associated with each problem type to be generated which controls the manner In 
which the data structure Is traversed. It also controls the probability of 
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Appendix 

Abstract Problems 



An abstract problem is a triple (dtita, unknown, relation) where data is a vec- 
tor of input variables. The unknown is the variable whose va 1 ue is sought as the 
solution. The relation defines a function which assigns a unique value to the 
unknown for each vector of data input values. An abstract problem can be repre- 
sented by a function whose domain corresponds to data, whose range corresponds 
to the unknown, and whose rule is given by the relation. 

o 

An interpretation I of an abstract problem consists of an associatioi of an 
object (s) to the abstract problem and thu assignment of specific attributes to 
tihe data variables and the unknown. 

An abstract problem can have many interpretations. The values of the attri- 
butes of the interpretation must belong to the domain" of the problem. 

Let Pj(I) denote the problem Pj with the interpretation I; unknown (Pj,I) 
denotes the solution to the problem Pj(I).__Let the class of problems generated 
(or represented) by Pi using all allowable interpretations be denoted Ci. 

Pi is a subabstract problem of Pj, denoted Pi - Pj, if Ci - Cj. f V 

Two abstract problems, Pi and Pj, are weakly equivalent if .Ci - Cj. 

Let Pi and Pj be abstract problems. Denote their domains by Di and Dj, 
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selectlng a more difficult problem primitive. This table Is updated as a 
student progresses In order to prevent too many similar problems from appearing, 

A grammar for a simple age problem and Its structure representation 
are given In Table 12 and Figure 7. The grammar shown Is only capable of 
generating a few sentences; the grammar actually used 13 considerably more 
complex The answer primitives shown are In a simple form which can be used 
to extract the solution to the problem (In the sense of writing the equations) 
but they currently do not supply enough Information to allow the program to lead 
the student from word problem to" equations. They will supply this information 
eventually but the final form is not set at this time. 

One could look at each rewrite ru le of the grammsr as a function which 
returns Its value In a manner similar to the manipulation generation routines. 
The reason for not doing so are technical characteristics of PL/ 1 Involving 
effective storage utilization and ease of mod If teat Ion. If one looks at the 
rewrite rules as functions which return values, the possibility of primitives 
which are evaluated for their effect rather than their value comes to mind. For 
example, placing a picture on the screen to supplement a geometrlc-t/pe word 
problem would Involve an entity which causes the figure to be displayed. The 
intention Is to experiment with this later. ' 

Given, the. above data structure, word problem generation Is relatively 
straight forward. The data structure Is scanned using the probabilities table 
to determine the relative difficulty of the problem, substitutions are made for 
variables In tie primitives (for instance, Ci and C2 In Table 12), and the 
solution equatlon(s) Is generated for the problem from the answer primitives. 
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D. Graphing Problem Generation 

The generation methods for graphing problems In a high school environment 
require little sophistication. Most problems are concerned with finding the 
slope, X-Y Intercept, or perhaps the points of Intersection of two first or 
second degree curves on a plane. The expression generator for the manipulation 
problems can be used to generate the equations to be graphed. A simple probabi- 
listic grammar can be used to decide the type of problem and the solution method. 
In short a combination of techniques from manipulation and word problem generation 
wll I be used. Pure graphing problems in this system will be a series of short, 
fast Interactions between machine and student in which the student receives immed- 
iate feedback after any mistakes. 

Graphing is a problem solving method and not an end in Itself and the Intention 
is to have graphics as available to the student for problem solving as algebraic 
manipulation. After the student has progressed- sufficiently, graphing may be 
the required solution method for word problems (particularly for problems with 
several equations and several unknowns) .For the final stages of a student's CAI 
work, he should be able to decide when to use graphics as the solution method for 
a problem. z- V 

Several general remarks about problem generation need to be made. In all three 
cases, the probability tables passed to the generating routines can control problem 
difficulty as well as problem type. In all three cases, an instructor experimenta- 
tion mode Is, available 'so that the instructor can experiment with the' 
probability tables and data structures to test the output of the generation routines. 
In the case of word problems, an additional difficulty can occur, particularly 
with a slow student. The number of possible problems is much smaller than for the 
other two types and for some concepts there may be only twc or three possible first 



-51- 

sentences. At the very least, there should be seme way to Insure that the 
number of repetitions Is minimal. (A problem Is certainly easier h it can 
be solved by direct analogy with a previous problem.) Therefore, Information 
concerning previous word problems will be associated with the student's record. 
The exact format is not set as yet. 

E. Solution Monitoring 

In a student's early stages of use, the system will lead him step by step 
through the solution. As a student progresses through the course his skills 
increase to the point where he no longer desires nor needs to be led through 
the problem. It Is, of course, much more difficult to monitor his work at this 
stage. The system can check that his work Is algebraically correct, start the 
problem over If the student desires, or in many cases suggest how to proceed to 
a solution. It Is d iff leu It- In complicated problems to keep the student from 
going off on a tangent. To circumvent this problem 1o some extent, a desk 
calculator mode for both manfpuletion and graphing is planned which is partial ly 
coded and debugged. 

This mode Is envisioned as being available to' the student from any point to 
allow experimentation (without machine supervision) In a sort of scratch paper- 
manner. For manipulation problems, commands to perform common manipulations will 
be vallable (e.g. 'ADD 3X' will add 3X to an expression). In the graphics 
mode, provision for plotting simple equations and connecting points under student 
"Control will be made, as well as allowing the student to change the scale of his 
coordinate system. 

This mode Is considered to be Important In developing a student's skills ' 
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wlth the problem solving tools of algebraic manipulation and graphing without 
too much computer Interference. The most difficult part of solution monitoring 
Is, of course, translation from a written problem description to the equations 
necessary to solve the problem. This process is only partially understood In 
humans and Is highly heuristic ( Paige, 1966). For this reason there Is not much the 
system can do for the student except to lead hfm through step by step In a 
very rigid manner. In the case of a more advanced student, the system can tell 
him whether his equations are right or. wrong and, perhaps, show htm one way to 
derive the correct equations. At this point, ft Is not clear how much can be 
done In this area and only actual experimentation will resolve this question. 
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Grammar for Algebra Word Problems 
TABLE 12 

CI = 5 | 6 | ... | 20 
C2 = CI + I8{ 19' . . . 1 30 
C4 = CI*C2 
C5 = C2-C I 
G3 = CI+C2 



PR0B= PI P|X|P2 P2X 
PiX = Pll Pi IX |FI2 PI2X 
PI IX = PIU|PII2 
PI2X = P 1 2 1 j P 1 22 
P2X = P2I P2IX|P22 P22X 
P2IX = P2I I |P2I2 
P22X = P22||P222 

PI = 'JACK IS C5 YEARS YOUNGER THAN HIS FATHER.' 
P2 ■ 'JACK'S AGE PLUS HIS FATHER'S AGE IS C5 YEARS.' 
PI I = 'THE SUM OF THEIR AGES I S C3.' ' 
PI2 = 'THE PRODUCT OF THEIR AGES IS C4. ' 
PHI = PI2I = 'HOW OLD IS JACK?' 
*P2| = 'THE DIFFERENCE OF THEIR AGES IS C5' * 
P22 = P|2 
P2I I = Rill = P22I 

P2I2 = P222 = 'HOW OLD ARE JACK AND HIS FATHER? ' 



A I = 'C5=CI-CI' 

A2 = 'CI+C2=C3' 

All = 'CI+C2=C3' 

AI2 = 'CI*C2=C4' 

AI2L= A!!!-= '?CI' - 

A2I = Al 

A22 = A 12 

A2 1 1 = A 1 1 1 * P22 1 

A2I2 - A222 =/?CI?C2' 
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■VII. Formal Model of Problem Generation and Solution 
A. Introduction 

Problem generation is described as originating from a semantic network 
(Quillian, 1969). This process is an extension of Carbonell's work to quanti- 
tative problem solving (Carbonell, 1970) and is motivated by Polya' s classic 
work on problems (Polya, 1945). Indeed, this research can be thought of as a 
step in the automation of Polya 's heuristics. Problem solution is closely related 
to problem generation. The plan for problem solution is generated at the same 
time that the problem is generated. 

In the above systems four basic elements can be discerned, memory, reasoning, 
input and output. Memory contains factual information such as course material 
and the student model. Reasoning usually takes the form of algorithms. It 
generates problems, solves problems, and makes decisions on the basis of input. 
If the reasoning component has learning capabilities it can-modify the contents 
of memory. These four elements at» e related according to the Block Diagram in 
Figure 8. 3 
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Block Diagram of Problem Generator/Solver 
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The form that each of these elements can be given is investigated in 

— - 

following sections. 

B# Memory and Problem Generation 

A general view of problems is taken. Let P be the class of all prob- 
lems tinder consideration. 

Any problem can be analyzed into three parts—the unknown, the data, 
and the condition (Polya, 1945). P is partitioned into subclasses by taking 
two problems to be equivalent if they are represented, by the same triple 
(data, unknown, relation); i.e., same without values and interpretation of 
variables. 

" The class of triples can also be grouped into equivalence classes 
according to problem type. Two expressions for a relation are equivalent 
if they represent ^the same relation. The -classification of problems can 
be .carried further by classifying relations according to various properties. 

The significance of these problem equivalence classes is that -all the 
problems in a class can be represented by their common triple. Moreover,- 
the system can select the unknown ^ from among data items." Hence, problem 
equivalence classes can be represented by the pain (data, relation). The 
data itself can be represented by (object, attribute, value) triples. Since 
the representations^ will not have values the final representation is 
(object(s), attributed), relation). = ^ 

The classifications- are summarized by the scheme, problem -* (data,^ 
unknown, relation) + (data, relation)"-*- (objebt(s^, attribute(s), relation) 
where this ^analysis proceeds from a specific instance of a problem to the 
= ^ most general representation, of the problem class. For generation these _ 
processes are reversed. A brief introduction to a formal theory of problem 
classification, generation and solution is presented in the Appendix. 

_K represents the system^ knowledge of the subject matter (at least 
a large part of M does) and can conveniently take the form of a semantic 
network.* "A semantic netowrk.is a graph structure with items corresponding 



to the nodes and relationships to the arcs. The items car. be objects, attri- 
butes, or pointers. Values are associated with the attributes. The relation- 
ships are semantic relationships if the semantic network is applied to natural 
language or logical relationships of interest to a particular subject matter. 
Carbonell and Wexler have demonstrated the utility of semantic networks for " : : 
storing factual information. Simmons (1970) has shown their utility for natural 
language analysis and generation. Melton (1971) has studied the automatic gener 
ation of semantic nets from text. Schwarcz (1370) has applied semantic nets to 
deductive question answering. Quillian (1969) has applied them to the comprehen- 
sion of English text. By extending a semantic net and basing it on a U tuple 
(object, attribute, value, relation), a semantic network can also be used for 
problem solving. . 

Semantic memories, then, can be used for several purposes, including storage 
of subject dependent information, natural language analysis and generation, prob 
lem -generation and solution. 

Problem generation with regard to a pair (data, relation) consists of 1), 
selection of = an. interpretation, 2) selection of an unknown, 3) generation of 
values for data attributes, (Steps 1 and 3 are an interpretation using the semen 
tic network as the domain of interpretation) selection of a particular rela- 
tion if a class of relations is specified in the pair. This generation process 
is the reverse ^of the prior problem dassif ication. 

In terms oir the model problem generation is performed by the operator R. 
R generates a problem as a function of memory M (the pairs (data, relation) and 
the student Is level). , 

- In -addition, if a semantic memory is used to represent, the subject informa- 
tion,, the system is also able to generate factual questions, evln for a subject 
which is oriented^ to problem solving. 

C. Problem Solution 

- In the model, R also corresponds to the problem solution mechanism. 

A pair (data, relation) is called an abstract problem and corresponds to a 
= concept(s)^in the concept tree. For solution, abstract problems or groups of = 
abstract problems can have associated solution -methods. 
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A problem solving system is described by a pair i?,S) ;;»u?e ? is a collection 
of abstract problems and S is a solution method for solving interpretations of P. 
For the problems of a particular subject area, suppose P r Pi U P2 U...U Pn. 
Since no one solution method may be practical for P, each Pi has an associated Si. 

A problem solving system needs some sort of planning mechanism with which to 
approach a problem. A plan is defined as some composition of Si f s. For a system 
which solves a restrictive class of problems the plan nood be no more than a fixed 
sequence of steps to be carried out. However, for a large class of problems the 
system will need plans that vary from complex problem to complex problem. A plan 
will indicate a decomposition of a complex problem into basic subproblems which 
belong to the Pi. Each subproblem has an associated solution method Si. A cor- 
responding composition of the solution methods will- yield a solution method for 
the complex problem. 

r Thus, the class of problems generated by abstract problems {Pi} is extended 
while keeping the same Si's. Various constructions are applied to the basis 
fPi}, yielding complex problems not in any of the classes Pi., Hopefully, the 
/new problems will be solvable using the Si's combined according to the type of 
construction used. Hence, if. (Pi U P 2 = 0...U Pn,{Si}) and U P 2 U...U Pn 

then C?, fSi}>, where Pi are the representations within the computer and t is the 
larger class of problems solvable by Si. 

Another way of extending a class is by transformation, i.e. suppose P is * 
solvable by S and T:P'-*P is a Transformation of P r to P. Under certain conditions 
R' is also solvable £>y S. These extension techniques are generation mechanisms 
:and also possible strategies for a planning mechanism. ^ , _ 

In a generative system the plan for problem solution can be a product of ^ 
problem ^generation.- The system constructs the problem cut of basic building 
blocks; i.e., abstract problems, each of which has an associated solution method. 
A corresponding construction applied to the solution methods comprises the plan 
for solving the constructed problem. 

Indeed, the implementation of the plan as, a set of procedures^ for solving 
the problem is a special case of automatic program synthesis (Manna,- 1970). The sol\ 
tion method is synthesized out the Si's. The solution procedure's control 
structure, expressed by the plan, will correspond ^to the particular construction 
^ used to compose the problem out of the subproblems Pi. 
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A flew diagram which summarizes the problem generation and solution pro- 
cess is shown in Figure S. ~ 

Problem Type 
(Data, relation) 
. (Data, unknown, relation) 

i 

i + 



Interpreted and 
evaluated triple^ 



Semantic ; 

net j 
Grammar i 



{Si} 
Solution 



Simple English expression 
cf the triple 

Figure 9 



Flow Diagram of Generation/Solution Process 
D. Input and Output 

Semantic networks can also express semantic relationships of objects. = In 
some applications, English has been mapped into a semantic net and related- 
information retrieved (Carbenell, 1970, Schwarcz, 1970h Semantic nets have 
likewise* been used for English generation ♦ 

^ Thus^ a semantic net can be used to convert an interpreted abstract problem 
to simple English statements — perhaps one for data, one for the unknown, one 
for the relation. 

For quantitative responses from the student, the system does not need 
sophisticated input conversion to tuples. Non-quantitative answers, however, 
would need conversion-to tlie intermediate form in order to make use of the 
semantic net. Such conversion is also necessary for the system to function as 
a^problem solver and question answerer which accepts English problems as input 
father than generating them, 

E. Learnings 

Learning by an intelligent system for CAI includes updating the model of 
the student, acquisition of new information, or revision of old information 
contained in memory. * 
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Learning can be rote learning or generalized learning (Samuel, 1967). Fur- 
ther, learning implies the acquisition of new knowledge. The new knowledge can 
be received by the system from the designer (its need having seen indicated by 
the system), obtained by the designer and the system, or acquired by the system 
itself. 

Learning, then, can be realized on several levels. Level 0 is not learning 
but is included for completeness. Level 0 occurs when the system designer inputs 
information or solution methods into the system. Level 1 occurs when the system 
has received a question or problem or generates one which it is unable to solve. 
The system remembers these and informs the designer who updates the semantic 
memory or increases the solving power of the system. Levels 0 and 1 are rote 
learning. Level 2 learning consists of solving a new type of problem by discovery 
(or selection) of similarititas or analogies with known problem classes. Level 3 
learning consists of solving a new type of problem by analyzing the = prcblem to 
discover its structure and then formulating a solution plan using the available 
solution methods. Level 2 and level 3~are generalized learning. Alitor these 
types of learning re suit in an increase in the question answering or problem 
solving power of a system. 

Another type of learning of concern for CAI is that of the student user. 
The CAI system should increase not only its own state of knov/ledge, but also that 
of the student user. 

Learning would, come directly into play- for a generative system if the process 
of generation were reversed and the system had to analyze a new problem to dis- 
cover its corresponding synthesis and solution. 
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is a partial order on abstract problems/ Pi is strongly equivalent to 
Pj if and only if Pi >.Pj and Pj > Pi* Also, if Pi > Pj then Pj - Pi. "> lf 
can be interpreted as more difficult than 

Various construction techniques for creating new abstract problems from 
given problems can be defined". 

Abstract problems and their relations have significance for problem genera- 
tion. It is possible to introduce solution methods into the discussion by 
attaching a predicate to an abstract problem. And, then, questions of signifi- 
cance of constructions on abstract problems for solution methods can be studied 
(Perry, 1973). 
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